Remove Basilisk from conformance results - #2330
Conversation
Remove Basilisk from conformance results Reverts python#2316, at the request of Basilisk's author. Some of Basilisk's rules match raw source text and hard-coded typing symbol names rather than resolved symbols on the AST, so its published score reflects the spelling of the test files rather than conformance to the specification. Semantics-preserving changes to the suite — renaming imported typing symbols, adjusting whitespace — collapse the result. Removes the basilisk result files, the BasiliskTypeChecker adapter, the dependency entries, and the README reference, and regenerates results.html from the remaining checkers. No other type checker's results are affected.
AlexWaygood
left a comment
AlexWaygood
left a comment
There was a problem hiding this comment.
Thank you! I really appreciate the honesty and self-reflection here.
I'm asking that you please leave the door open to make another attempt in future. We still have full confidence that Basilisk is a worthy type checker.
I can't speak for the other maintainers of this repo, but I certainly would have no problem with this, assuming the underlying issues here are addressed
|
honestly i don't understand why a 100% vibecoded type checker was even treated as a serious project in the first place. and since even this apology looks AI generated i don't see why the door should be left open if there doesn't even appear to be any remorse from a human at all. is the developer even aware that this PR was made? |
|
@DetachHead (my personal opinion, not speaking for the typing council) a) The PR to add Basilisk was accepted in the first place because this situation had never happened before, and thus we didn't have any existing guidelines around what qualifies a tool to be added to the conformance suite, beyond "it can run the conformance suite and someone submitted a PR to add it". This situation has made it clear that we need those guidelines and a bit more due diligence before we accept such pull requests in future. b) IMO the whole "door open" thing is a bit meaningless / a given. I guess we could choose to ban this author from ever submitting something to the conformance suite again, but IMO that would be an over-reaction. We are going to be more careful in future about adding any new type checker, and that certainly includes (probably to an even greater degree) any future re-submission of Basilisk. |
Without participating in the typing community and understanding the problems people are trying to solve, you tell AI to loop until the conformance tests are passing then go around bragging and talking smack about other tools on social media. Is that what you call trust-building? 🤡 |
5 participants
Reverts #2316. I'm the author of Basilisk, and I'm asking for it to be taken out of the conformance results. Our published score doesn't measure what the table says it measures, so it needs to be removed.
Why
Many of Basilisk's rules match against raw source text and hard-coded typing symbol names instead of resolved symbols on the AST. The result is that our score reflected the exact spelling of the test files rather than conformance to the specification.
A mutation harness makes this unambiguous. It renames imported typing symbols and adjusts whitespace. It should not move any checker's score. Ours moves. I reproduced this myself and 113 of 141 tests are affected. We will investigate exactly how much work is necessary to fix this.
How it happened
Despite our best efforts to enforce accuracy, we relied too heavily on AI to validate the work we did, and we focused too heavily on conformance results instead of thoroughly testing real world scenarios. It looks as though we didn't pick up code that clearly should have only been just a placeholder and that code proliferated. This should have been predictable and I take full responsibility for letting this happen.
I am sorry for breaking trust, especially when the whole aim of Basilisk as a tool is to build trust. We have an absolute commitment to building the trust of the Python community so requesting this removal is a necessary step until we can work through the issues.
Resubmission
I'm asking that you please leave the door open to make another attempt in future. We still have full confidence that Basilisk is a worthy type checker.
Before we submit again, we'll hold ourselves to the test that proved the issues. If a rename or a whitespace change moves our score, the rule isn't implemented and the number doesn't count. We'll publish that measurement alongside any resubmission, and if the honest score is lower, we'll publish the lower score.
We will do a full audit to remove any code that pattern-matches source text. We will track it publicly in our repo. We will take as long as it takes to get this right and not submit again until we've got proof that we're not making the same mistakes again.
Sincere apologies to the maintainers here. You accepted our submission in good faith and our carelessness led to misleading results being posted here.